Robot control system and robot control method

ABSTRACT

To control articulated robots by dynamically modifying a combination of a hardware-dependent middleware layer and a hardware-independent application layer.  
     An interface and a database for semantically performing operation are prepared between a middleware layer which depends upon the hardware configuration of a robot and an application layer which does not depend upon the hardware configuration, thereby making it possible to always guarantee normal operation even if a combination of the middleware and the application which is to be introduced onto the robot is modified. The application can acquire appropriate input data via the middleware, and can issue an appropriate command.

TECHNICAL FIELD

[0001] The present invention relates to a robot control system forcontrolling articulated robots, such as legged walking robots, using asoftware program, and more particularly, it relates to a robot controlsystem for controlling articulated robots, in which the hardwareconfiguration might be significantly modified according toattachment/detachment or replacement of moving units such as legs and ahead, using a software program. More specifically, the present inventionrelates to a robot control system for controlling articulated robotsusing a software program which comprises a combination of a softwarelayer having a high dependency upon the hardware configuration and asoftware layer having no dependency upon the hardware configuration, andto a program interface between the software layers, and moreparticularly, it relates to a robot control system for controllingarticulated robots by dynamically modifying a combination of ahardware-dependent software layer, such as middleware, and ahardware-independent software layer, such as an application, and to aprogram interface between the software layers.

BACKGROUND ART

[0002] A mechanical apparatus which utilizes electric or magneticactions to perform motions which resemble motions of human beings isreferred to as a “robot.” It is said that the word robot isetymologically derived from the Slavic word ROBOTA (slave machine). Inour country, robots have been widely used since the end of the 1960s,but most of them have been manipulators for the purpose of automated orunmanned production operations at factory or industrial robots such asconveyor robots.

[0003] Recently, research and development have advanced on the structureand stable walk control of legged mobile robots including pet robotswhich simulate the physical mechanisms or motions of four-legged walkinganimals such as dogs and cats, and “human shaped” or “human type” robots(humanoid robots) which simulate the physical mechanisms or motions ofbiped walking animals such as human beings or apes. Thus, expectationson the practical use thereof have increased. These legged mobile robotsare less stable and have more difficult posture control and walk controlthan crawler robots, but are advantageous in that they can realizeflexible walking and running motions such as moving up and down thestairs and leaping over obstacles.

[0004] Stationary type robots, like arm type robots, which are installedand used in a particular place, perform activities only in fixed andlocal working spaces such as part assembling or selecting jobs. On theother hand, mobile robots, whose working spaces are not restrictive,freely move on a predetermined path or out of a path to take onpredetermined or arbitrary human operations or to offer various serviceson behalf of human beings, dogs, or other living things.

[0005] One use of legged mobile robots is to take on various difficulttasks in industrial activities, production activities, etc. For example,dangerous jobs or difficult jobs such as maintenance jobs in nuclearpower plants, thermal power plants, or petrochemical plants, partconveying and assembling jobs at manufacturing factory, cleaning of tallbuildings, and rescues at fires or other sites are taken on, etc.

[0006] Other uses of legged mobile robots include living uses, i.e.,“coexistent” uses with human beings or “entertainment” uses. Robots ofthis type emulate a variety of emotional expressions using the motionmechanisms or the extremities of relatively intelligent legged walkinganimals such as human beings and dogs (pets). Not only are pre-enteredmotion patterns strictly performed, but vivid expressions whichdynamically respond to words or attitudes (such as “praise,” “scolding,”“hitting,” etc.) received from a user (or any other robot) are alsodemanded.

[0007] Conventional toy machines have a fixed relationship between auser operation and a responsive motion, and the motions of the toycannot be modified according to the preference of users. As a result,users soon get tired of such toys that repeat the same motions.

[0008] On the other hand, intelligent robots comprise a behavior modelor a learning model which originates from motions, and allows the modelsto be modified based on external input information, such as voices,pictures, or touch, to determine a motion, thereby realizing anautonomous thought or motion control. Robots which are provided with anemotion model or an instinct model can develop autonomous behaviorsaccording to the robots' own emotions or instinct. The robots areequipped with an image input apparatus or voice input/output apparatusto perform an image recognition process or a voice recognition process,thereby also making it possible to realize realistic communication withhuman beings at a higher intelligence level.

[0009] Recent legged mobile robots have high information processingability, and these intelligent robots themselves can be thus consideredas a kind of computing system.

[0010] For example, robots maintain models of various rules for motions,such as an emotion model, a behavior model, and a learning model.According to each of the models, the robots make a behavior plan inresponse to external factors such as a user's action, and perform thebehavior plan by driving jointed actuators or through voice outputs,which can be then fed back to the user. A motion control of the robotsfor making a behavior plan or performing it on the machine isimplemented in the form of executing a program code (for example, anapplication etc.) on the computing system.

[0011] A major difference between a general computing system and a robotis that the former has fewer differences in the kind or combination ofhardware components constituting the system (that is, hardwareconfiguration) from system to system, while the latter has a hardwareconfiguration which significantly varies from system to system. Forexample, there are a variety of kinds of mobile robots, including arobot having movable units attached to a body formed of a head, legs,and a tail, and a robot consisting of a body and a wheel.

[0012] In a computing system in which the installed hardwareconfiguration is relatively uniform from system to system, the design ofsoftware executed on the system may not be relatively affected byhardware. On the contrary, in of the latter robot case, particularly, acontrol software layer, such as middleware in which hardware operationis executed, has an extremely high dependency upon hardware.

[0013] For example, if a moving control of a robot is considered, thecriteria of determining the stability during moving and walking iscompletely different whether moving means comprises movable legs, or awheel, or two legs or four legs, and an operating environment in whichthe application is executed is significantly different from system tosystem.

[0014] If the software development of robots is considered, in view ofthis circumstance, it seems efficient to discriminate a software layerhaving a relatively low dependency upon hardware from a software layerhaving a high dependency upon hardware. In other words,hardware-independent software and hardware-dependent software areseparately developed, and a combination thereof is modified to provide aproduct lineup having a variety of different characteristics andcapabilities.

[0015] The hardware-independent software is, for example, applicationlayer software in which processing which is less associated withhardware operations such as an emotion model, a behavior model, and alearning model is performed. The hardware-dependent software is, forexample, middleware layer software formed of a group of software moduleswhich provide basic features of a robot 1, and the configuration of eachof the modules is affected by hardware attributes including themechanical or electrical characteristics or specifications and the shapeof the robot. Roughly, the middleware is functionally classified intorecognition middleware which processes and recognizes an input of asensor of each portion, and then notifies the upper application of this,and by output middleware which performs a control to drive hardware,such as driving of jointed actuators, according to the commands issuedby the application.

[0016] For example, the same application is executable on robots havingdifferent hardware configurations by introducing middleware suitable forthe hardware configurations into the robots.

[0017] Meanwhile, the manner in which software is introduced intovarious computing systems represented by robots can include supplyingnew software via a removable medium, and downloading software over anetwork. For example, a memory slot in which a removable medium such asa memory card or a memory stick is loaded is provided in a portion onthe robot body, thereby only requiring a task of inserting/removing theremovable medium to/from the slot in order to readily introduce newsoftware such as an application or middleware into the robot.

[0018] To introduce new software into a control system of a robotcomprising a plurality of software layers, the newly introduced softwaremust be kept at a good compatibility, i.e., must be compatible, with theother software layers.

[0019] More specifically, in order to permit any combination of anapplication and middleware, a format for which data or commands areexchanged between the software layers, that is, an interface betweenprograms, must be established.

DISCLOSURE OF INVENTION

[0020] An object of the present invention is to provide a superior robotcontrol system for controlling articulated robots, such as leggedwalking robots, using a software program.

[0021] Another object of the present invention is to provide a superiorrobot control system for controlling articulated robots, in which thehardware configuration might be significantly modified according toattachment/detachment or replacement of moving units such as legs and ahead, using a software program.

[0022] Another object of the present invention is to provide a superiorrobot control system for controlling articulated robots using a softwareprogram which comprises a combination of a software layer having a highdependency upon the hardware configuration and a software layer havingno dependency upon the hardware configuration, and to provide a programinterface between the software layers.

[0023] Another object of the present invention is to provide a superiorrobot control system for controlling articulated robots by dynamicallymodifying a combination of a hardware-dependent software layer, such asmiddleware, and a hardware-independent software layer, such as anapplication, and to provide a program interface between the softwarelayers.

[0024] The present invention has been made in view of the foregoingproblems, and a first aspect thereof provides a robot control system forcontrolling motions of a robot which comprise a combination of aplurality of hardware components including:

[0025] a first control unit for performing a process which does notdepend upon hardware configuration information of the robot;

[0026] a second control unit for performing a process which depends uponthe hardware configuration information of the robot; and

[0027] a communication unit for providing communication between thefirst and second control units.

[0028] As used herein, “system” refers to a logical group of a pluralityof apparatuses (or function modules which perform specific functions),and does not particularly refer to the fact that the apparatuses orfunction modules are accommodated in a single housing or not.

[0029] The first control unit used herein is implemented by anapplication layer software which does not depend upon the hardwareconfiguration. The second control unit is implemented by middlewarelayer software having a high dependency upon the hardware configuration.The communication unit is mounted in the form of a program interfacewhich realizes a data exchanging process between the application and themiddleware.

[0030] According to the robot control system in the first aspect of thepresent invention, the communication unit is an interface between theapplication layer and the middleware layer, and the format for whichdata or commands are exchanged between the software layers isestablished, so that an arbitrary combination of the application and themiddleware can be permitted.

[0031] The first control unit is realized by, for example, executingapplication software which determines a behavior sequence of the robotusing a model in which the a configuration or motion of the robot isabstracted. The application software includes an emotion model whichmodels an emotion of the robot, an instinct model which models theinstinct, a learning model which sequentially stores a causalrelationship between an external event and a behavior taken by therobot, a behavior model which models a behavior pattern, etc., by way ofexample.

[0032] The second control unit is realized by executing middlewaresoftware which provides a basic onboard function of the robot. Themiddleware software is formed of, for example, a recognition processorunit which receives input data detected from hardware of the robot via asystem control layer to detect external factors such as distancedetection, posture detection, and contact, taking account of thehardware configuration, and an output processor unit for processing anonboard motion control for the robot based on a command from theapplication.

[0033] The communication unit notifies the first control unit of theinformation detected by the recognition processor unit, and transfersthe command of the first control unit to the output processor unit.

[0034] The communication unit includes an information communicationinterface which aids in notifying the first control unit of theinformation from the second control unit, command interface which aidsthe first control unit in controlling the second control unit, etc.

[0035] The communication unit may include an information database inwhich the first control unit semantically designates information to beretrieved from the second control unit. In such a case, a target recordin the information database is registered, so that target informationcan be transferred from the second control unit to the first controlunit.

[0036] The communication unit may include a command database in whichthe first control unit semantically designates a command to be issued tothe second control unit. In such a case, the first control unit can usethe command database to semantically select a command.

[0037] The communication unit may include a feedback interface whichaids in notifying the second control unit of the recognition result ofthe first control unit, and which aids in notifying the first controlunit of the relationship between the recognition result and the behaviorfeasible in the second control unit.

[0038] The first control unit and the second control unit may bestructured so as to be capable of being independently handled.

[0039] The communication unit may notify the first control unit of asystem event detected by the second control unit.

[0040] The communication unit may also include means for notifying saidfirst control unit of a shutdown factor which is detected by said secondcontrol unit, and means for notifying said second control unit of aresume condition with respect to the shutdown which is set by said firstcontrol unit. It may further include means for notifying said firstcontrol unit of a recommended resume condition which is set by saidsecond control unit.

[0041] According to the robot control system in accordance with thefirst aspect of the present invention, an interface and a database forsemantically performing operation are prepared between a middlewarelayer which depends upon the hardware configuration of a robot and anapplication layer which does not depend upon the hardware configuration,thereby making it possible to always guarantee normal operation even ifa combination of the middleware and the application which is to beintroduced onto the robot is modified. The application can acquireappropriate input data via the middleware, and can issue an appropriatecommand.

[0042] A second aspect of the present invention provides a robot controlmethod for controlling motions of a robot which comprise a combinationof a plurality of hardware components using a first control module forperforming a process which does not depend upon hardware configurationinformation of the robot, and a second control module for performing aprocess which depends upon the hardware configuration information of therobot,

[0043] the robot control method including a communication step ofproviding communication between the first control module and the secondcontrol module.

[0044] As used herein, the first control module is implemented byapplication layer software which does not depend upon the hardwareconfiguration. The second control module is implemented by middlewarelayer software having a high dependency upon the hardware configuration.The communication step can be implemented in the form of a programinterface which realizes a data exchanging process between theapplication and the middleware.

[0045] According to the robot control method in the second aspect of thepresent invention, in the communication step, an interface between anapplication layer and a middleware layer is realized to establish theformat for which data or commands are exchanged between the softwarelayers, thereby permitting an arbitrary combination of the applicationand the middleware.

[0046] The first control module is implemented by application softwarewhich determines a behavior sequence of the robot using a model in whicha configuration or motion of the robot is abstracted. The applicationsoftware is formed of, for example, an emotion model which models anemotion of the robot, an instinct model which models the instinct, alearning model which sequentially stores a causal relationship betweenan external event and a behavior taken by the robot, a behavior modelwhich models a behavior pattern, etc.

[0047] The second control module is implemented by middleware softwarewhich provides a basic onboard function of the robot. The middleware isformed of, for example, a recognition processor module which receivesinput data detected from hardware of the robot via a system controllayer to detect external factors such as distance detection, posturedetection, and contact, taking account of the hardware configuration,and an output processor module for processing an onboard motion controlfor the robot based on a command from the application.

[0048] In the communication step, the first control module may benotified of the information detected by executing the recognitionprocessor module, and the command by executing the first control modulemay be transferred to the output processor module.

[0049] In the communication step, an information communication interfacewhich aids in notifying said first control module of the informationfrom the second control module may be used.

[0050] In the communication step, a command interface which aids thefirst control module in controlling the second control module may beused.

[0051] In the communication step, an information database in which thefirst control module semantically designates information to be retrievedfrom the second control module may be used. In such a case, a targetrecord in the information database is registered, so that the targetinformation can be transferred from the second control module to thefirst control module.

[0052] In the communication step, using a command database in which thefirst control module semantically designates a command to be issued tothe second control module, the first control module may semanticallyselect the command.

[0053] In the communication step, a feedback loop may be executed tonotify the second control module of the recognition result of the firstcontrol module, and to notify the first control module of therelationship between the recognition result and the behavior feasible inthe second control module.

[0054] The first control module and the second control module may bestructured so as to be capable of being independently handled.

[0055] The communication step may include a substep of notifying thefirst control module of a system event detected by the second controlmodule.

[0056] The communication step may include a substep of notifying thefirst control module of a shutdown factor which is detected by thesecond control module, and a substep of notifying the second controlmodule of a resume condition with respect to the shutdown which is setby the first control module. The communication step may further includea substep of notifying the first control module of a recommended resumecondition which is set by the second control module.

[0057] According to the robot control method in the second aspect of thepresent invention, an interface and a database for semanticallyperforming operation are prepared between a middleware layer whichdepends upon the hardware configuration of a robot and an applicationlayer which does not depend upon the hardware configuration, therebymaking it possible to always guarantee normal operation even if acombination of the middleware and the application which is to beintroduced onto the robot is modified. The application can acquireappropriate input data via the middleware, and can issue an appropriatecommand.

[0058] A third aspect of the present invention provides a robot controlsystem which is configured by an object-oriented program, including:

[0059] an application object for executing a process which does notdepend upon a hardware configuration of a robot;

[0060] a middleware object for executing a process which depends uponthe hardware configuration of the robot;

[0061] an information database registered with information which is usedfor said middleware object and which corresponds to a semantic commandfrom said application object; and

[0062] object control means for controlling communication between saidapplication object and said middleware object on the basis of saidinformation database.

[0063] The information database may hierarchically describe a semanticaspect of the registered information. Preferably, the format of saidinformation database at least includes an information identificationinformation field, a classification field, and a sensor informationidentification field.

[0064] Other objects, features, and advantages will be apparent from amore detailed description of the following embodiments of the presentinvention or in conjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0065]FIG. 1 is a view schematically showing a hardware configuration ofa robot in accordance with an embodiment of the present invention.

[0066]FIG. 2 is a view showing an example modification of theconfiguration of a driving subsystem 50 by attaching/detaching orreplacing CPC components.

[0067]FIG. 3 is a view showing an example modification of theconfiguration of a driving subsystem 50 by attaching/detaching orreplacing CPC components.

[0068]FIG. 4 is a view showing an example format of an informationdatabase.

[0069]FIG. 5 is a view showing an example entry of records in theinformation database.

[0070]FIG. 6 is a view showing an example entry of records in theinformation database.

[0071]FIG. 7 is a view schematically showing the manner how informationused by an application is registered, and the manner how the registeredinformation is announced through middleware.

[0072]FIG. 8 is a view showing the mechanism in which an applicationobject requests sensor value information to a middleware object.

[0073]FIG. 9 is a chart schematically showing communication between theobjects in order for the application object to acquire the sensor valueinformation.

[0074]FIG. 10 is a view schematically showing the data structure of thesensor value information.

[0075]FIG. 11 is a view schematically showing the data structure ofheader information.

[0076]FIG. 12 is a view schematically showing the data structure ofsensor information (SensorInfo).

[0077]FIG. 13 is a view showing the manner how a sensor is convertedinto the position on the reference coordinate by Tmatrix.

[0078]FIG. 14 is a view showing the manner how the application objectissues a command according to command information which is announcedthrough a feedback loop using a feedback interface.

[0079]FIG. 15 is a view schematically showing the data structure oftarget information (TargetInfo) which is used in the feedback loop.

[0080]FIG. 16 is a view schematically showing the data structure ofTargetStatus indicating the status of a target.

[0081]FIG. 17 is a view schematically showing the data structure of Size3D indicating the size of the target in a rectangular block form.

[0082]FIG. 18 is a view schematically showing the data structure of avariable Enable 3D in FIG. 17.

[0083]FIG. 19 is a view schematically showing the data structure fcommand information (CommandInfo) which is used in the feedback loop.

[0084]FIG. 20 is a view schematically showing the data structure ofStatus indicating a relationship with the target.

[0085]FIG. 21 is a view schematically showing processing of a systemevent via the application interface.

[0086]FIG. 22 is a view schematically showing the data structure of asystem event (SysEvent).

[0087]FIG. 23 is a view showing an example usage of aid.

[0088]FIG. 24 is a view showing an example usage of aid.

[0089]FIG. 25 is a view showing example processes for announcing ashutdown factor via the application interface and for setting a resumecondition.

[0090]FIG. 26 is a view schematically showing the data structure ofshutdown factor PowerEvent which the middleware object notifies theapplication object of.

[0091]FIG. 27 is a view schematically showing the data structure ofshutdown factor PowerEvent which the middleware object notifies theapplication object of.

[0092]FIG. 28 is a view showing an example format of a command database.

[0093]FIG. 29 is a view showing an example entry of records in thecommand database.

[0094]FIG. 30 is a view showing an example entry of records in thecommand database.

[0095]FIG. 31 is a view showing the mechanism in which a command isissued and the status is announced between the application object andthe middleware object.

[0096]FIG. 32 is a view schematically showing, on a chart, communicationbetween the objects to execute the command issued by the applicationobject.

[0097]FIG. 33 is a view schematically showing the data structure of thecommand issued by the application object.

[0098]FIG. 34 is a view schematically showing the data structure ofTarget 1D.

[0099]FIG. 35 is a view schematically showing the data structure ofTarget 1D.

[0100]FIG. 36 is a view schematically showing a data improvement ofresource information (ResourceInfo).

BEST MODE FOR CARRYING OUT THE INVENTION

[0101] Hereinbelow, embodiments of the present invention will bedescribed in detail with reference to the drawings.

[0102]FIG. 1 schematically illustrates the hardware configuration of arobot in accordance with an embodiment of the present invention. Asshown in the same figure, hardware of the robot is formed of acontrolling subsystem 10 and a driving subsystem 50.

[0103] The controlling subsystem of the robot is constituted by a CPU(Central Processing Unit) 11, a main memory 12, a fixed memory device13, and a removable memory device 14.

[0104] The CPU 11 as a main controller is adapted to comprehensivelycontrol operation of the overall apparatus, i.e., the robot, byexecuting hardware-independent program, such as an application, or ahardware-dependent program, such as middleware, under a control ofsystem control software.

[0105] The CPU 11 is connected via a bus to the memory or other circuitcomponents or peripherals. A unique address (memory address or I/Oaddress) is allocated to each of the devices on the bus, and the CPU 11can communicate with a specific device on the bus by addressing it. Thebus is a common signal transmission path including an address path, adata path, and a control path.

[0106] The main memory 12 typically comprises a volatile storage deviceincluding a plurality of DRAM (Dynamic Random Access Memory) chips, andis used to load execution program codes of the CPU 11 or to temporarilysave the work data thereof. In the present embodiment, a program codesuch as an application or middleware, which is supplied from the fixedmemory device 13 or the removable memory device 14, is loaded onto themain memory 12, that is, mapped onto the memory space.

[0107] The fixed memory device 13 is a non-replaceable and non-volatilestorage device which is fixedly mounted to the robot body. For example,like a flash memory, by applying a write voltage, a programmablenon-volatile memory element may be used to constitute the fixed memorydevice 13.

[0108] The fixed memory device 13 is used to store a program code suchas an application for controlling the motion or thought of the robot andmiddleware for hardware operation. Since the fixed memory device 13 isfixedly installed to the apparatus, preferably, a version ofhardware-dependent software such as middleware, which supports thehardware configuration at time of delivery of the robot (default) or thestandard hardware configuration, is prepared in the fixed memory device13.

[0109] The removable memory device 14 is a non-volatile storage devicewhich is attachably/detachably or replaceably mounted to the robot. Forexample, the removable memory device 14 is formed using a cartridge typestorage medium such as a memory card or a memory stick, and it is loadedinto a predetermined memory slot for replaceable use on the machine.

[0110] As in the fixed memory device 13, the removable memory device 14is used to store a program code such as an application for controllingmotions or thoughts of the robot and middleware for hardware operation.However, the removable memory device 14 is provided so as to beattached/detached to/from or replaced with respect to the robot body,and can be used when up-to-date software is provided to the machinesince it is expected that it moves from machine to machine havingdifferent hardware configurations for use.

[0111] There is lower necessity to store hardware-dependent software,such as middleware, in the removable memory device 14, in particular,with the consciousness of whether or not it is a version supporting thehardware configuration at time of delivery of the robot (default) or thestandard hardware configuration. Rather, preferably, the middlewarecapable of providing an operating environment to the hardwareconfiguration assumed by the application is stored in the removablememory device 14 together with the application.

[0112] On the other hand, the driving subsystem 50 of the robot includesjointed actuators, driving control circuits thereof, an encoder fordetecting motions, and various kinds of sensors such as a camera andcontact sensors (all the components are not shown). In the example shownin the figure, the driving subsystem 50 is handled per unit of drivingunits such as the head, the body, and the legs.

[0113] Furthermore, at least a portion of the driving units is formed asa physical component (CPC: Configurable Physical Component) which can bedynamically reconfigured by attaching/detaching or replacing it to/fromor with respect to the machine.

[0114] In the present embodiment, unique identification information,that is, a component ID, is allocated to each of the physicalcomponents. The CPU 11 (more specifically, system control softwareexecuted on the CPU 11) in the controlling subsystem 10 accesses each ofthe mounted physical components via the bus to transfer a controlcommand to each of the physical components or to acquire the respectivecomponent IDs. A detected combination of the component IDs correspond tocurrent hardware configuration information of the robot.

[0115]FIG. 2 illustrates an example modification of the configuration ofthe driving subsystem 50 by attaching/detaching and replacing the CPCcomponents. In FIG. 2(a), a plurality of physical components such as thehead, the legs, and the tail are attached to the body (that is, thehardware configuration information is {body ID, head ID, leg ID, tailID}), and in FIG. 2(b), only a wheel is attached to the body as aphysical component (that is, the hardware configuration information is{body ID, wheel ID}).

[0116] As in the example shown in FIG. 2(a) and FIG. 2(b), the samehardware-dependent software cannot be used in robots havingsignificantly different hardware configurations. For example, themiddleware which performs a sensor input from the head or the tailcannot run on the apparatus having the hardware configuration shown inFIG. 2(b). Similarly, the middleware which is designed so that the legsare drivingly controlled as moving means cannot be used on the apparatushaving the hardware configuration shown in FIG. 2(b).

[0117] Next, the configuration of the robot control software will bedescribed with reference to FIG. 3.

[0118] The robot control software is formed of an application layerwhich does not depend upon the hardware configuration of the robot, amiddleware layer which depends upon the hardware configuration, and adevice driver at the bottom layer. The software of each layer controlsthe hardware of the robot, that is, the driving subsystem 50 asdescribed later, under a control of a predetermined operating system(OS).

[0119] In the present embodiment, the design of the software of eachlayer may employ object-oriented programming. Each object-orientedsoftware is handled in the unit of module of “object” which integratesthe data with the procedure associated with the data.

[0120] Data is communicated between the application layer and themiddleware layer via a predetermined programming interface (hereinafterreferred to as “application interface”). In the present embodiment, avariety of combinations of the application and the middleware arepermitted by introducing the software via the removable memory device14, etc. Each compatibility of the application with the middleware isrealized by the application interface. The mechanism of the applicationinterface is described later in detail.

[0121] The application layer and the middleware layer are each formed ofa plurality of object files. In the present embodiment, the applicationlayer and the middleware layer are under a control of an objectmanagement system. Data is communicated between the middleware layer andthe device driver at the bottom layer via a predetermined programminginterface (hereinafter referred to as “device driver interface”).

[0122] The hardware operations, such as an issuance of control commandsto jointed actuators for driving and an entry of detection values insensors, are performed not directly by the middleware but via therespective device drivers corresponding to the hardware components.

[0123] The application comprises an emotion model which models emotionsof the robot, an instinct model which models the instinct, a learningmodel which sequentially stores causal relationships between externalevents and behaviors taken by the robot, and a behavior model whichmodels behavior patterns, so that an output destination of the behaviordetermined by the behavior model based on the sensor input information,i.e., the external factors, can be changed.

[0124] The emotion model and the instinct model have a recognitionresult and a behavior record at inputs to manage an emotion value and aninstinct value, respectively. The behavior model can refer to theemotion value and the instinct value. The learning model updates thebehavior selection probability based on the teaching of the learningfrom the outside (operator) to provide the updated contents to thebehavior model.

[0125] The application performs a calculation using a model in whichconfigurations or motions of the robot are abstracted, and is thushardware-independent software which is not affected by hardwareattributes.

[0126] The middleware layer comprises a group of software modules whichprovide basic onboard functions of the robot, and the configuration ofeach of the modules includes hardware-dependent software which isaffected by hardware attributes including mechanical or electricalcharacteristics or specifications, and the shape of the robot.

[0127] The middleware layer can be functionally classified intorecognition middleware and output middleware.

[0128] The recognition middleware receives raw data from the hardware,such as image data, audio data, and other detection data obtained fromthe sensors, via the system control layer, and processes them. That is,processes such as voice recognition, distance detection, posturedetection, contact, motion detection, and color recognition areperformed based on the various input information to obtain recognitionresults (for example, ball was detected, tumbling was detected, stroked,hit, sounds of do-mi-so were heard, a moving object was detected, anobstacle was detected, an obstacle was recognized, etc.). Therecognition result is sent to the upper application layer, and is usedto make a behavior plan, etc.

[0129] On the other hand, the output middleware provides functionsincluding walking, reproduction of motions, synthesis of output sounds,and a control to emit LEDs that are located correspondingly to eyes.That is, in response to a command which is associated with the behaviorplan made by the application layer, a servo instruction value or outputsound, emitted light (LEDs), and output voice of joints of the robot aregenerated for each of the functions, and are played on the robot via anoutput, namely, a virtual robot. With such a mechanism, abstractbehavior commands (for example, to advance, to recede, to be pleasant,to bark, to sleep, to do exercise, to be surprised, to track, etc.) aregiven from the application side, thereby making it possible to controlthe motions by the jointed actuators of the robot or other onboardoutput units.

[0130] The middleware according to the present embodiment preparessemantic information communication between the application and themiddleware, and can share an intelligence base. The database contains aninformation database (informationDB) which is used by the application tosemantically designate input information, and a command database(CommandDB) which is used by the application to semantically select anexecution command. The detail of the databases will be described later.

[0131] The software of each robot control layer is supplied to theinside of the machine of the robot 1 by the fixed memory device 13 orthe removable memory device 14. The software of each layer is loadedonto the main memory 12, namely, is mapped on the memory space, for use.

[0132] In the present embodiment, the application interface isinterposed and defined between the middleware layer which depends uponthe hardware configuration of the robot, and the application layer whichdoes not depend upon the hardware configuration such as behaviorselection. Therefore, the following functions can be satisfied.

[0133] (1) The middleware and the application are structured so as to beindependently handled, so that a single application can support robotshaving different hardware configurations (a user can move his owned (orbred) application from machine to machine or can distribute or sell it).

[0134] (2) The middleware and the application are structured so as to beindependently handled, so that middleware supporting variousapplications can be created on a machine having a single hardwareconfiguration. The reusability of the middleware is enhanced, and thedevelopment efficiency of the application is therefore improved.

[0135] (3) The middleware and the application are structured so as to beindependently handled, so that improvements of the actual expressioncapability and the controllability of the same application may beachieved on a machine having the same hardware configuration bymodifying the middleware.

[0136] (4) Since binary compatibility is guaranteed, the above-describedadvantages can be enjoyed by downloading binary codes of the applicationor the middleware from the removable memory device 14 such as a memorystick to the machine to easily replace software. In addition, softwaremay be exchanged without any compiling work.

[0137] (5) Since the application and the middleware are independent,software developing vendors can devote themselves to functiondevelopment of their own fields. For example, a vendor which specializesa control system concentrates on the development of middleware, so thata variety of applications may be used.

[0138] The application interface according to the present embodimentcomprises function implementation means as follows.

[0139] (1) An information communication interface(SensorInformation+SensorValueVector/SystemInfo/Boot/Shootdown) whichaids in notifying the application of information detected by themiddleware.

[0140] (2) A command interface (Command/Status) which aids theapplication in controlling the middleware.

[0141] (3) A feedback interface (TargetInfo/CommandInfo) which aids innotifying the middleware of the recognition result detected by theapplication, and which aids in notifying the application of therelationship between the recognition result and the behavior feasible inthe middleware.

[0142] (4) Means for semantically selecting information used by theapplication among the data supplied by the information communicationinterface.

[0143] This semantically selecting means allows the application toacquire input information (for example, “stroked” by a user, etc.) forlearning without the consciousness of the actual location of a sensor asan information source. If expected hardware such as sensors and switcheswere not present onboard, alternative sensors could be selected (forexample, some switch disposed on the head is alternatively used toperceive as being “stroked,” etc.) and associated with correspondingemotion (for example, pleasant emotion).

[0144] The middleware prepares an information database InformationDB (asdescribed later) to semantically designate input information. Theapplication registers information used for InformationDB, so thatsemantic acquisition of the input information can be realized.

[0145] (5) Means for semantically selecting a command to be executed onthe application among the commands supported by the command interface.

[0146] This semantically selecting means allows the application toexecute a necessary behavior without the consciousness of whichmotor/effector is to be actually activated by the middleware.

[0147] The middleware is provided with a command database CommandDB (asdescribed later) to semantically realize a command selection.

[0148] (6) Means for providing robot information, necessary forrecognition by the application, which dynamically changes duringexecution.

[0149] For example, a HeaderInfo field (as described later) is definedin SensorInformation to provide information on the posture or motionspeed of the robot.

[0150] (7) Means for providing sensor physical information, necessaryfor recognition by the application, which dynamically changes duringexecution.

[0151] For example, a HeaderInfo field in the SensorInformation, and aSensorInfo field are used to announce which direction the sensor isoriented in.

[0152] (8) Means for providing physical information, such as the sizeand motion speed of the robot, which is necessary to realize a movingbehavior using the application.

[0153] For example, information on whether or not the robot having thecurrent hardware configuration can pass through a space detected, etc.,is provided to a TargetPos·application layer of Command.

[0154] (9) Means for announcing a previously prepared behavior, which isassociated with the information input from the information communicationdata, using a command interface in order to constitute a reflection loopwhich does not depend upon the structure.

[0155] Classes of an Interaction system are prepared to search CommandDBusing SensorID of the input information as Key.

[0156] (10) Means for asking the application to execute a commandaccording to the convenience of the middleware.

[0157] For example, by making it possible to handle a process, such as arecovery operation from tumbling, which changes depending upon theconfiguration of the middleware, without need for the application layerto understand the detail, the statuses of the middleware and theapplication can be matched.

[0158] Additionally speaking with respect to tumbling, a method forrecovering from the tumbling posture, i.e., through what posture it cansafely stand, varies depending upon the kind of posture that issupported by the middleware. Of course, this also depends upon thecapabilities of the middleware.

[0159] (11) Means for announcing a shutdown (Shutdown) and a resume(Resume) condition.

[0160] Shutdown factors and resume conditions are closely influenced.For example, if the temperature of a battery during discharging is toohigh, the time required for the temperature to drop to an appropriatetemperature depends upon the characteristics of the machine such as aheat dissipation capability of hardware. As the resume condition at thistime, the middleware notifies the application of the time at which itmay occur next as a recommended resume condition. The application setsthe final resume condition from the recommended condition and atconvenience of the application itself, and notifies the middleware of itat time of shutdown. (Shutdown/Resume Condition)

[0161] (12) Means for utilizing the recognition result of theapplication under a control of the middleware.

[0162] By adding the recognition result, output service, to the Trackingcommand in the application layer, a software module which handlesTracking of the middleware can acquire the recognition result designatedfrom the recognition result service without intervention of the softwarewhose command has been determined at an application layer, therebyincreasing the speed of processing. (TargetID of TargetInfo/Command)

[0163] (13) Means for allowing the system to automatically search aparameter, which may be modified according to the hardwareconfiguration, to notify the middleware of this.

[0164] For example, even new hardware configuration would not requiresoftware modification, but only require replacement of parameter files.The system may sometimes search a ROM mounted on the physical componentsand announce the information.

[0165] The function provided by this means improves versatility, whichalso influences binary compatibility.

[0166] Hereinafter, representative function implementation means of theapplication interface is described in detail.

[0167] Information Database (InformationDB):

[0168] In the present embodiment, information used by the applicationcan be semantically selected in the data supplied via the applicationinterface. For this reason, the middleware prepares an informationdatabase. The application registers the used information in theinformation database in a semantic basis, thereby making it possible toacquire necessary input information.

[0169] The application can acquire input information (for example,“stroked” by a user, etc.) for learning without the consciousness of theactual location of a sensor as an information source. If expectedhardware such as sensors and switches were not present onboard,alternative sensors could be selected (for example, some switch disposedon the head is alternatively used to perceive as being “stroked,” etc.)and associated with corresponding emotion (for example, pleasantemotion).

[0170]FIG. 4 shows an example format of the information database. Asshown in the same figure, one record in the information database isformed of an information identification information (InformationID)field, a classification field, a sensor value information field, and asensor identification information (SensorID) field.

[0171] The information identification information (InformationID) is aunique identifier which is uniquely allocated to a record, and can bedetermined at the discretion of the middleware.

[0172] The classification field is further divided into subfields ofcompatibility, major, intermediate, and minor. The compatibilitysubfield is a flag indicating that the ID written subsequently theretois meant by either standard or extended ID. A major classification, anintermediate classification, and a minor classification are filled inthe major, intermediate, and minor subfields, respectively.

[0173] With the consciousness of semantic selection of the information,that is, the classification is such that the semantic aspect of theinformation is hierarchically described. By selecting the necessaryinformation after the database is read, the operation using acombination of different databases is possible. The minor subfield is inthe form of the actual hardware configuration (the unit system etc. aredetermined at this time).

[0174] The sensor value information field includes subfields in whichthe maximum value, the minimum value, and the resolution of a targetsensor input are filled.

[0175] The sensor identification information (Sensor ID) field includessubfields in which sensor parts and sites on which the sensors aremounted are designated, and flags indicating compatibility of the partsand sites.

[0176] The classification and the sensor identification information arekeys to select a target record when the database is searched.

[0177] Software such as the application searches the informationdatabase of the middleware which is combined at time of execution toretrieve necessary information identification information, andinformation is extracted from the information group which is announcedduring operation using the information identification information as asearch key to apply necessary processing.

[0178]FIGS. 5 and 6 illustrate example entries of record in theinformation database.

[0179] In each of the figures, the records of the information IDs 101and 102 substantially designate the same sensor. Similarly, the recordsof 200 and 201 substantially designate the same sensor. More detailedsite information is described in 102 than in 101. 201 is different from200 in that a relationship between the location measured by a PSD andthe location captured by a camera is obtained as an image-associatedtype.

[0180] The application uses this information database to register inputinformation to be used, thereby making it possible to receiveannouncement of the target sensor input information from the middleware.

[0181]FIG. 7 schematically shows the manner how an application objectregisters information to be used, and the manner how the applicationobject is notified of the registered information via a middlewareobject. Hereinbelow, the mechanism in which the application objectregisters the used information and the mechanism in which the registeredinformation is received by the application object will be described withreference to the same figure.

[0182] The application object issues a semantic database request of“stroke” to the object management system, for example, when sensor valueinformation on “stroke” is to be used (T1).

[0183] In response to this database request, the object managementsystem searches the information database using the semantic class of“stroke” as a search key. Then, the target record is found, and theinformation identification information thereof is returned to theapplication object of the requesting source (T2).

[0184] The application object make a declaration (namely, registration)to the object management system that this information is used if thereturned value is satisfied (T3), and notifies the middleware object ofthe ready state, namely, that an input of the target sensor valueinformation is acceptable (T4). In response to the declaration, theobject management system makes a registration confirmation to themiddleware object (T5).

[0185] In response to the registration confirmation, the middlewareobject instructs the driver object which performs a hardware operationof the sensor designated by the target record in the informationdatabase to activate the sensor (that is, sensor open) (T6).

[0186] As a result of sensor open, the middleware object can entersensor value information at any time (T7). The middleware objectcollectively announces the input information from the sensors for eachapplication object according to the registered contents (T8).

[0187] Sensor Value Information (Sensor Information):

[0188] The mechanism in which the application object uses theinformation database to register the used information to enable thedesired sensor value information to be input has been described above.Next, the mechanism in which the application object acquires the sensorvalue information, and the detail of the sensor value information aredescribed.

[0189]FIG. 8 illustrates the mechanism in which the application objectrequests the sensor value information to the middleware object.

[0190] The application object can acquire information identificationinformation (Information ID) based on the information database to obtainnecessary information.

[0191] The application object uses this information identificationinformation to issue a data read (Read) request to the middlewareobject. In response to this Read request, the middleware object notifiesthe application object of the sensor value information input via thedriver object.

[0192]FIG. 9 schematically shows, on a chart, communication between theobjects when the application object acquires the sensor valueinformation.

[0193] As shown in the same figure, the application object issues theRead request to the middleware object when it requires the sensor valueinformation.

[0194] The middleware object receives sensor values from the sensors viathe driver object. In response to the Read request from the applicationobject, it returns the sensor value associated with the requestedinformation identification information to the application object.However, the sensor values which are input when the application objectis not ready are not transferred to the application object, but arediscarded.

[0195]FIG. 10 schematically illustrates the data structure of the sensorvalue information. As shown in the same figure, the sensor valueinformation is formed of a header and a body.

[0196] The header is constituted by header information (HeaderInfo) andsensor information (SensorInfo). Information of the type which cannot bedescribed in the information database and which changes during motionscan be added to the header.

[0197] Since a plurality of pieces of information are delivered at onetime, the target header can be searched for using the informationidentification information. An offset to value information is writteninto the retrieved header. The value comprises a data stream, and thetarget sensor value can be extracted from the header information and thesensor information.

[0198]FIG. 11 schematically illustrates the data structure of the headerinformation. As shown in the same figure, the header information isconstituted by header size (HeaderSize), body size (BodySize), thenumber of sensors (NumOfSensor), time (Time), robot speed (RobotSpeed),and posture identification information (PostureID).

[0199] The robot speed (RobotSpeed) has translation speed components androtation speed components written in each of the x, y, and z axes, asshown in the same figure.

[0200] The posture identification information (PostureID) uses what isdefined as the common concept. The posture identification information isdefined as the following data structure.

[0201] TypeDef PostureID Byte

[0202]FIG. 12 schematically shows the data structure of the sensorinformation (SensorInfo). The format of the body can be described in thesensor information (SensorInfo).

[0203] The sensor information (SensorInfo) includes informationidentification information (information ID), and information foraccessing information defined from the body by classification. Theinformation for accessing the body is constituted by the amount ofoffset of the body (offset), the vertical size (vertical), and thehorizontal size (horizontal), and the number of skips (skip), the numberof steps (step), etc.

[0204] otime indicates the time at which this sensor value informationwas detected, and is affected by a delay time of a filter, etc. Limit isa flag indicating whether or not the sensor itself cannot move since itis located at the end point.

[0205] Tmatrix is information indicating the status of the sensor. Morespecifically, it is a 4×4 matrix used to convert the sensor coordinateinto the reference coordinate. The manner how the sensor is convertedinto the position on the reference coordinate by Tmatrix is shown inFIG. 13.

[0206] Feedback Interface:

[0207] The application interface according to the present embodimentachieves a feedback loop through which the middleware is notified of therecognition result detected by the application, and through which theapplication is notified of the relationship between the recognitionresult and the behavior feasible in the middleware. An interface for thefeedback loop, namely, “feedback interface” is mounted.

[0208]FIG. 14 illustrates the manner how the application object issues acommand according to the command information which is announced throughthe feedback loop using the feedback interface.

[0209] The middleware object transmits sensor value information such asimage data to the application object based on the registered contents ofthe specification information.

[0210] In response, the application object recognizes a predeterminedtarget based on the sensor value information, and returns the targetinformation (TargetInfo) in a object form as a recognition result to themiddleware object.

[0211] Furthermore, the middleware object feeds command information(CommandInfo) concerning a possible interaction with the target back tothe application object.

[0212] The application object selects a selectable command based on thecommand information, and issues the command to the middleware object.The middleware object performs a predetermined hardware operation viathe driver object so that the command can be implemented on the robotmachine.

[0213]FIG. 15 schematically shows the data structure of the targetinformation (TargetInfor) used in the feedback loop.

[0214] As shown in the same figure, the target information isconstituted by Time indicating the time at which the information wasdetected, AppID used to identify the target to be tracked, SensorID usedto identify the sensor used, Tmatrix which is a conversion matrix fromthe reference coordinate to the sensor coordinate, TargetStatusindicating the status of the target (either Unknown, Lost, or NotSupportstatus is taken) (see FIG. 16), TargetPos indicating, in the form ofpolar coordinate, the position of the target in the sensor coordinatesystem, and Size 3D indicating the size of the target in a rectangularblock form (see FIG. 17; with respect to the data structure of avariable Enable 3D in FIG. 17, see FIG. 18).

[0215]FIG. 19 schematically shows the data structure of the commandinformation (CommandInfo) used in the feedback loop.

[0216] As shown in the same figure, the command information isconstituted by Time indicating the time at which the information wasdetected, AppID used to identify the target to be tracked, SensorID usedto identify the sensor used, Status indicating a relationship with thetarget (see FIG. 20), Category indicating the command of the interaction(that is, selectable command), and Position 3D indicating a relationshipbetween the target and the center.

[0217] A command in a Kick-Touch system varies in a different mannerdepending upon the configuration of the robot. Tracking, etc., arehandled as fixed commands.

[0218] The command information shown in FIG. 19 is of the type whichprovides versatility of selection using Category. An examplemodification can include the type which provide no versatility,represented by a CommandID stream, in which ActionID is designated.

[0219] System Event:

[0220] The application interface according to the present embodimentcomprises a mechanism which, when a predetermined event generated on therobot machine, namely, a system event, is detected, notifies theapplication object of this. In response to the notification of thesystem event, the application object executes a predetermined command.FIG. 21 illustrates the processing of the system event via theapplication interface.

[0221]FIG. 22 schematically shows the data structure of the system event(SysEvent) which is used by the middleware object to notify theapplication object of the event.

[0222] As shown in the same figure, the system event is constituted byTime indicating the time at which the event occurred, Categoryindicating a command of the interaction (that is, selectable command),Naction indicating the number of actions, and ActionID indicated by aCommandID stream.

[0223] The process corresponding to the system event can be allocated toan array type variable aid[i] constituting the action. An example usageof aid is shown in FIG. 23 and FIG. 24.

[0224] The system event does not include the header information(HeaderInfo) or the sensor information (SensorInfo), unlike the sensorvalue information.

[0225] Furthermore, when the application object is not ready, the systemevent is not discarded but is accumulated.

[0226] Shutdown/Resume:

[0227] The application interface according to the present embodimentcomprises a mechanism in which the middleware object notifies theapplication object of factors of shutdown (Shutdown), and a mechanism inwhich the application object sets conditions to resume (Resume), thatis, to subsequently boot, the middleware object.

[0228] The shutdown factors and the resume conditions are closelyinfluenced. For example, if the temperature of a battery duringdischarging is too high, the time required for the temperature to dropto an appropriate temperature depends upon the characteristics of themachine such as a heat dissipation capability of hardware. As the resumecondition at that time, the middleware object notifies the applicationof the time at which it may occur next as a recommended resumecondition. The application object sets the final resume condition fromthe recommended resume condition and at convenience of the applicationobject itself, and notifies the middleware of it at time of shutdown.

[0229]FIG. 25 illustrates an example process to announce the shutdownfactor and to set the resume condition via the application interface.

[0230] For example, when PowerEvent in which the temperature of abattery during discharging is too high is detected as a shutdown factor,the middleware object notifies the application object of PowerEvent.When the middleware object announces PowerEvent, it also announces thetime at which it may occur next, the resume condition, as a recommendedresume condition (RecommendCondition).

[0231] In response, the application object sets the final resumecondition (ResumeCondition) from the recommended resume condition and atconvenience of the application object itself, and notifies themiddleware object of this at time of shutdown.

[0232]FIG. 26 schematically shows the data structure of the shutdownfactor PowerEvent which the middleware object notifies the applicationobject of.

[0233] As shown in the same figure, PowerEvent is constituted by Timeindicating the time at which the shutdown factor occurred, Categoryindicating a command of the interaction (namely, selectable command),RecommendCondition indicating a condition recommended as a resumecondition, Naction indicating the number of actions, and ActionIDindicated by a CommandID stream.

[0234]FIG. 27 schematically shows the data structure of ResumeConditionwhich is transferred by the application object to the middleware objectin order to set a next boot condition.

[0235] As shown in the same figure, ResumeCondition is constituted byMask indicating a mask at time of notification of the event, Eventindicating an event at time of shutdown, and ResumeTime for setting thenext resume time.

[0236] Command Database (CommandDB):

[0237] The application interface according to the present embodimentprovides a command interface which aids the application object incontrolling the middleware object. The middleware object prepares acommand database in order to make it possible to semantically select acommand to be executed on the application object among the commandssupported by the command interface. The application object can execute anecessary behavior without the consciousness of which motor/effector isactually activated by the middleware object.

[0238]FIG. 28 shows an example format of the command database. As shownin the same figure, one record in the command database is constituted bya command identification information (CommandID) field, a classificationfield, a resource field, a posture information field, a sensoridentification information (SensorID) field, an infinite execution flagfield, a generation identification information (generation ID) field,and a degree identification information (degree ID) field.

[0239] The command identification information (CommandID) is a uniqueidentifier which is uniquely allocated to a record, and can bedetermined at the discretion of the middleware.

[0240] The classification field is further divided into subfields ofcompatibility, major, intermediate, and minor. The compatibilitysubfield is a flag indicating that the ID written subsequently theretois meant by either standard or extended ID. A major classification, anintermediate classification, and a minor classification are described inthe major, intermediate, and minor subfields, respectively. With theconsciousness of semantic selection of the information, that is, theclassification is such that- the semantic aspect of the information ishierarchically described.

[0241] The resource field is further divided into bit subfields permanagement unit of motion (the head, tail, and legs), sound (a singlespeaker), and specialization (a head LED, and a tail LED). Allocation ofthe meaning of bit values of the bit subfields can be changed dependingupon the capability of the middleware.

[0242] The posture information field is further divided into subfieldsin which a start posture and an end posture during command execution,and posture variations (the posture may change even if the postures atthe beginning and at the end are the same) are designated.

[0243] The infinite execution flag is a flag indicating that there is noend of execution.

[0244] The sensor identification information (SensorID) field includessubfields in which sensor parts to be used when the command is executedand sites on which the sensors are mounted are designated, and flagsindicating the compatibility of the parts and the sites.

[0245] The generation identification information (generationID) field isa field which expresses using a bit value of the corresponding bitposition whether or not it is a command available to generations such asadults, children, and infants. For example, 011 indicates that it can becommonly available to children and infants.

[0246] The degree identification information (degree ID) field is afield in which an index value indicating the nominal degree of thecommand (for example, the amount of movement) is filled.

[0247] The maximum generation number, the maximum degree, etc., areindividually defined from the shown records.

[0248] The command database contains a header (not shown), and, ifmeeting it, the meaning of extension is known and can be used.

[0249] Software such as the application object searches the commanddatabase of the middleware object which is combined during execution,and can select the behavior to be executed according to the currentinstinct or emotion or any other predefined scenario. The commandidentification information allocated to the selected behavior is passedto the middleware object to execute that behavior, thereby realizing theintended behavior on the robot machine.

[0250] The application object can enhance the expression ability on therobot machine by selecting the command so that the resources do notoverlap. Furthermore, in the state where the robot is idle, the commandwhose infinite execution flag has been set is selected to make itpossible to continue some action.

[0251]FIG. 29 and FIG. 30 show example entries of the records in thecommand database.

[0252] The records of command IDs 100, 101, and 102 indicate commandswhich all express “pleasant.” The application object can select any ofthe commands according to the statuses of the robot machine (forexample, the degree of exhaustion, generation, etc.), and designatewhich record of “pleasant” is to be used.

[0253] The records of command IDs 200 and 201 are commands indicatingthat an object kicks. They can be used when it is known which motion,sound, or LED that behavior corresponds to, and when the behavior isreliably desired to execute.

[0254] Command Status:

[0255] The application interface according to the present embodimentprovides a command interface which aids the application object incontrolling the middleware object. In response to a command issued bythe application object, the middleware object executes the command onthe robot machine via an agent such as a driver object, and returns thestatus to the application object.

[0256]FIG. 31 illustrates the mechanism of an issuance of a command andan announcement of the status between the application object and themiddleware object.

[0257] The application object can acquire command identificationinformation (CommandID) of a command to realize an action satisfying thecurrent instinct or emotion, or any other predefined scenario based onthe command database.

[0258] The application object issues a command when the middlewareobject is ready. In response, the middleware object executes the commandon the robot machine via an agent such as a driver object, and returnsthe status to the application object.

[0259]FIG. 32 schematically shows, on a time chart, communicationbetween the objects to execute the command issued by the applicationobject.

[0260] The application object issues a command to the middleware object.In response, the middleware object locks the resource and the posture tomake the use by any other command unauthorized, and returns the resourceand posture locked information to the application object as the status.Thereafter, the middleware object notifies the application object of thefact that it is ready.

[0261] The middleware object requests an agent such as a driver objectto execute the command. The input sensor information, etc., are returnedfrom the driver object.

[0262] Once the execution of the command has completed, the middlewareobject cancels the locked state for the next command execution, andreleases the resource. It also returns the status including the resourcereleasing information and the posture information of the machine to theapplication object.

[0263] When the application object successively issues commands at Ntimes or infinite times, basically, the middleware object also returnsthe status in a similar manner to the procedure in the circumstancewhere a command is issued once. The resource releasing information andthe posture information are supplied to the application object beforethe final command is executed by the driver object, thus allowing theapplication object to know the status of the resource and to know a nextselectable range of the command.

[0264]FIG. 33 schematically illustrates the data structure of thecommand issued by the application object.

[0265] As shown in the same figure, the command is constituted by MW-IDdesignating a command ID, TargetID designating subject identificationinformation (SubjectID), and TargetPos indicating, in the form of polarcoordinate, the position of the target in a sensor coordinate system(however, whereas TargetPos of TargetInfo is a sensor center, inTargetPos as used herein, the applied coordinate is a coordinate atwhich the sensor site is fixed to the body).

[0266] MW-ID is a field for designating command identificationinformation (CommandID) used. The command which can be individuallydesignated includes three kinds, i.e., “normal,” “emergency stop,” and“stop.”

[0267] The normal command is overwritten. The commands concerning soundor LEDs are the same as the case of the emergency stop. The motion isthe same as the case of the stop. The emergency stop does not guaranteethe stable state or the posture. The stop corresponds to a stop at thestable state or posture, and is basically equivalent to the case whereit waits for a normal termination.

[0268]FIG. 34 schematically illustrates the data structure of TargetID.If 0ID contained in SubjectID is 0×0, TargetPos is enabled.

[0269]FIG. 35 schematically illustrates the data structure of the statusreturned by the middleware object to the application object. As shown inthe same figure, the status is constituted by MW-ID designating thecommand ID, CommandStatus indicating the status of the command,ActionStatus indicating the status of the action, posture identificationinformation PostureID, and resource information ResourceInfo.

[0270] The command status CommandStatus is defined for three states,i.e., “Complete,” “Not Support,” and “Incomplete.” If there is nocommand to be executed or in case of emergency stop, it becomes“Incomplete.”

[0271] The action status ActionStatus is defined for three states, i.e.,“Success” (when success of execution was detected), “Fail” (when failureof execution was detected), and “Unknown” (when there is no function todetermine the execution).

[0272] The resource information ResourceInfo describes the resourceinformation used by the middleware. FIG. 35 schematically shows the datastructure of ResourceInfo. As shown in the same figure, ResourceInfo hasresources of motion, sound (speaker), LED, etc., described in the formof bitmap. The correspondence between the bits and the resources can befreely defined. FIG. 36 schematically illustrates a data improvement ofthe resource information (ResourceInfo) The posture identificationinformation PostureID describes the current posture.

[0273] Appendix

[0274] The present invention has been described with reference to thespecific embodiment. However, it is apparent that those skilled in theart may make modifications or alternatives of the embodiment withoutdeparting from the scope of the present invention.

[0275] The gist of the present invention is not necessarily limited toproducts called “robot.” In other words, any mechanical apparatus whichuses electric or magnetic actions to move similarly to the motions ofhuman beings could equally be implemented by the present invention evenif it is a product which falls within other industrial fields such astoys.

[0276] In summary, the present invention has been disclosed in anillustrative form, and is not intended to be construed as restrictive.In order to define the gist of the present invention, the section ofCLAIMS noted at the beginning of the Description should be referred to.

Industrial Applicability

[0277] According to the present invention, a superior robot controlsystem is provided which controls articulated robots, such as leggedwalking robots, using a software program.

[0278] Furthermore, according to the present invention, a superior robotcontrol system is provided which controls articulated robots, in whichthe hardware configuration might be significantly modified according toattachment/detachment or replacement of moving units such as legs and ahead, using a software program.

[0279] Furthermore, according to the present invention, a superior robotcontrol system for controlling articulated robots using a softwareprogram which comprises a combination of a software layer having a highdependency upon the hardware configuration and a software layer havingno dependency upon the hardware configuration, and a program interfacebetween the software layers are provided.

[0280] Furthermore, according to the present invention, a superior robotcontrol system for controlling articulated robots by dynamicallymodifying a combination of a hardware-dependent software layer, such asmiddleware, and a hardware-independent software layer, such asapplications, and a program interface between the software layers areprovided.

[0281] According to the present invention, a program interface between amiddleware layer which depends upon the hardware configuration of arobot and an application layer which does not depend upon the hardwareconfiguration is established, so that normal operation can always beguaranteed even if a combination of the middleware and the applicationwhich is to be introduced onto the robot is modified.

[0282] According to the present invention, an interface and a databasefor semantically performing operation are prepared between applicationand middleware, so that the application can acquire appropriate inputdata via the middleware, and can issue an appropriate command forhardware operation to the middleware.

1. A robot control system for controlling motions of a robot whichcomprise a combination of a plurality of hardware components,comprising: a first control unit for performing a process which does notdepend upon hardware configuration information of the robot; a secondcontrol unit for performing a process which depends upon the hardwareconfiguration information of the robot; and a communication unit forproviding communication between said first and second control units. 2.A robot control system according to claim 1, wherein said first controlunit is realized by executing application software which determines abehavior sequence of the robot using a model in which a configuration ormotion of the robot is abstracted.
 3. A robot control system accordingto claim 1, wherein said first control unit comprises at least one of anemotion model which models an emotion of the robot, an instinct modelwhich models the instinct, a learning model which sequentially stores acausal relationship between an external event and a behavior taken bythe robot, and a behavior model which models a behavior pattern.
 4. Arobot control system according to claim 1, wherein said second controlunit is realized by executing middleware software which provides a basiconboard function of the robot.
 5. A robot control system according toclaim 1, wherein said second control unit includes a recognitionprocessor unit which receives input data detected from hardware of therobot via a system control layer to detect external factors such asdistance detection, posture detection, and contact, and an outputprocessor unit for processing an onboard motion control for the robotbased on a command from said first control unit.
 6. A robot controlsystem according to claim 5, wherein said communication unit notifiessaid first control unit of the information detected by the recognitionprocessor unit, and transfers the command of said first control unit tothe output processor unit.
 7. A robot control system according to claim1, wherein said communication unit comprises an informationcommunication interface which aids in notifying said first control unitof the information from said second control unit.
 8. A robot controlsystem according to claim 1, wherein said communication unit comprises acommand interface which aids said first control unit in controlling saidsecond control unit.
 9. A robot control system according to claim 1,wherein said communication unit comprises an information database inwhich said first control unit semantically designates information to beretrieved from said second control unit, and a target record in theinformation database is registered to transfer target information fromsaid second control unit to said first control unit.
 10. A robot controlsystem according to claim 1, wherein said communication unit comprises acommand database in which said first control unit semanticallydesignates a command to be issued to said second control unit, and saidfirst control unit uses the command database to semantically select acommand.
 11. A robot control system according to claim 1, wherein saidcommunication unit comprises a feedback interface which aids innotifying said second control unit of the recognition result of saidfirst control unit, and which aids in notifying said first control unitof the relationship between the recognition result and the behaviorfeasible in said second control unit.
 12. A robot control systemaccording to claim 1, wherein said first control unit and said secondcontrol unit are structured so as to be capable of being independentlyhandled.
 13. A robot control system according to claim 1, wherein saidcommunication unit notifies said first control unit of a system eventdetected by said second control unit.
 14. A robot control systemaccording to claim 1, wherein said communication unit comprises: meansfor notifying said first control unit of a shutdown factor which isdetected by said second control unit; and means for notifying saidsecond control unit of a resume condition with respect to the shutdownwhich is set by said first control unit.
 15. A circuit according toclaim 13, wherein said communication unit further includes means fornotifying said first control unit of a recommended resume conditionwhich is set by said second control unit.
 16. A robot control method forcontrolling motions of a robot which comprise a combination of aplurality of hardware components using a first control module forperforming a process which does not depend upon hardware configurationinformation of the robot, and a second control module for performing aprocess which depends upon the hardware configuration information of therobot, said robot control method comprising a communication step ofproviding communication between the first and second control modules.17. A robot control method according to claim 16, wherein the firstcontrol module is implemented by application software which determines abehavior sequence of the robot using a model in which a configuration ormotion of the robot is abstracted.
 18. A robot control method accordingto claim 16, wherein the first control module comprises at least one ofan emotion model which models an emotions of the robot, an instinctmodel which models the instinct, a learning model which sequentiallystores a causal relationship between an external event and a behaviortaken by the robot, and a behavior-model which models a behaviorpattern.
 19. A robot control method according to claim 16, wherein thesecond control module is implemented by middleware software whichprovides a basic onboard function of the robot.
 20. A robot controlmethod according to claim 16, wherein the second control module includesa recognition processor module which receives input data detected fromhardware of the robot via a system control layer to detect externalfactors such as distance detection, posture detection, and contact, andan output processor module for processing an onboard motion control forthe robot based on a command from the first control module.
 21. A robotcontrol method according to claim 20, wherein in said communicationstep, the first control module is notified of the information detectedby executing the recognition processor module, and the command byexecuting the first control module is transferred to the outputprocessor module.
 22. A robot control method according to claim 16,wherein in said communication step, an information communicationinterface which aids in notifying said first control module of theinformation from the second control module is used.
 23. A robot controlmethod according to claim 16, wherein in said communication step, acommand interface which aids the first control module in controlling thesecond control module is used.
 24. A robot control method according toclaim 16, wherein in said communication step, an information database inwhich the first control module semantically designates information to beretrieved from the second control module is used to register a targetrecord in the information database, thereby transferring the targetinformation from the second control module to the first control module.25. A robot control method according to claim 16, wherein in saidcommunication step, using a command database from which the firstcontrol module semantically designates a command to be issued to thesecond control module, the first control module semantically selects thecommand.
 26. A robot control method according to claim 16, wherein insaid communication step, a feedback loop is executed to notify thesecond control module of the recognition result of the first controlmodule, and to notify the first control module of the relationshipbetween the recognition result and the behavior feasible in the secondcontrol module.
 27. A robot control method according to claim 16,wherein the first control module and the second control module arestructured so as to be capable of being independently handled.
 28. Arobot control method according to claim 16, wherein said communicationstep includes a substep of notifying the first control module of asystem event detected by the second control module.
 29. A robot controlmethod according to claim 16, wherein said communication step comprises:a substep of notifying the first control module of a shutdown factorwhich is detected by the second control module; and a substep ofnotifying the second control module of a resume condition with respectto the shutdown which is set by the first control module.
 30. A robotcontrol method according to claim 29, wherein said communication stepfurther comprises a substep of notifying the first control module of arecommended resume condition which is set by the second control module.31. A robot control system which is configured by an object-orientedprogram, comprising: an application object for executing a process whichdoes not depend upon a hardware configuration of a robot; a middlewareobject for executing a process which depends upon the hardwareconfiguration of the robot; an information database registered withinformation which is used for said middleware object and whichcorresponds to a semantic command from said application object; andobject control means for controlling communication between saidapplication object and said middleware object on the basis of saidinformation database.
 32. A robot control system according to claim 31,wherein said information database which hierarchically describes asemantic aspect of the registered information.
 33. A robot controlsystem according to claim 31, wherein the format of said informationdatabase at least includes an information identification informationfield, a classification field, and a sensor information identificationfield.